Papers with scoring function
Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs (2025.findings-naacl)
Copied to clipboard
Duygu Nur Yaldiz, Yavuz Faruk Bakman, Baturalp Buyukates, Chenyang Tao, Anil Ramakrishna, Dimitrios Dimitriadis, Jieyu Zhao, Salman Avestimehr
| Challenge: | Existing methods for probability-based UE are limited by their inability to handle biased probabilities and complex semantic dependencies between tokens. |
| Approach: | They propose a learning-based scoring function that captures complex dependencies between tokens and probabilities and produces more reliable responses. |
| Outcome: | The proposed function outperforms existing scoring functions in question-answering and arithmetical reasoning tasks with different datasets. |
Improving Multilingual Instruction Finetuning via Linguistically Natural and Diverse Datasets (2024.findings-emnlp)
Copied to clipboard
Sathish Reddy Indurthi, Wenxuan Zhou, Shamil Chollampatt, Ravi Agrawal, Kaiqiang Song, Lingxiao Zhao, Chenguang Zhu
| Challenge: | Advancements in Large Language Models (LLMs) have significantly enhanced instruction-following capabilities, but most IFT datasets are predominantly in English, limiting model performance in other languages. |
| Approach: | They propose a method for collecting multilingual IFT datasets that preserves linguistic naturalness and ensures prompt diversity. |
| Outcome: | Experiments show that LLMs fine-tuned using this method show significant improvements in generative and discriminative tasks. |
RelWalk - A Latent Variable Model Approach to Knowledge Graph Embedding (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for embedding entities and relations in knowledge graphs are heuristically motivated and theoretical understanding of such embeddables is underdeveloped. |
| Approach: | They extend the random walk model of word embeddings to Knowledge Graph Embeddings (KGEs) they propose a learning objective motivated by the theoretical analysis to learn KGEs from a given knowledge graph. |
| Outcome: | The proposed learning objective is motivated by the theoretical analysis to learn KGEs from a given knowledge graph. |
Debiased Orthogonal Boundary-Driven Efficient Noise Mitigation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing noise mitigation methods face limitations due to their task-specific design, model dependency, and significant computational overhead. |
| Approach: | They propose a model-agnostic noise label mitigation paradigm that uses an estimator model and a scoring function to assess the noise level of input pairs. |
| Outcome: | The proposed method is superior to existing methods in training tasks and tasks. |
MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs (2024.acl-long)
Copied to clipboard
Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, Salman Avestimehr
| Challenge: | Generative Large Language Models (LLMs) are widely utilized for their excellence in various tasks. however, their tendency to produce inaccurate or misleading outputs poses a potential risk. |
| Approach: | They propose a new scoring function that considers the semantic contribution of each token in the generated sequence in the context of the question. |
| Outcome: | The proposed scoring function improves UE performance on a medical QA dataset. |
Towards a Better Metric for Evaluating Question Generation Systems (D18-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics based on n-gram similarity do not correlate well with human judgments . large datasets for document Question Answering (QA) have enabled the development of end-to-end supervised models . |
| Approach: | They propose a scoring function to capture answerability of questions . they also integrate existing similarity metrics into the scoring function . |
| Outcome: | The proposed scoring function improves human judgments on question answerability . the proposed scoring functions are made publicly available . |
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown promising results in zero-shot settings due to surface form competition . since probability mass is finite, this lowers the probability of the correct answer . |
| Approach: | They propose a scoring function that compensates for surface form competition by reweighing each option according to its a priori likelihood. |
| Outcome: | The proposed scoring function achieves consistent gains in zero-shot over calibrated and uncalibrated scoring functions on all GPT-2 and GPT-3 models on a variety of multiple choice datasets. |
Iterative Edit-Based Unsupervised Sentence Simplification (2020.acl-main)
Copied to clipboard
| Challenge: | Sentence simplification is relevant in various real-world and downstream applications. |
| Approach: | They propose an edit-based approach to unsupervised sentence simplification that uses a scoring function to score fluency, simplicity, and meaning preservation to perform edits. |
| Outcome: | The proposed model is more controllable and interpretable than state-of-the-art models on newsela and WikiLarge datasets. |
Experience as Source for Anticipation and Planning: Experiential Policy Learning for Target-driven Recommendation Dialogues (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to enhance dialogues lack the ability to anticipate user interactions . current approaches lack the capability to anticipate past interactions and to neglect past experiences . |
| Approach: | They propose a framework for enhancing dialogue anticipation with an experiential scoring function that estimates dialogue state potential using similar past interactions stored in long-term memory. |
| Outcome: | Experiments on two datasets show the framework is superior and effective . tree-structured EPL assesses past dialogue states with LLMs and MCTS . |
PiCSAR: Probabilistic Confidence Selection and Ranking for Reasoning Chains (2026.findings-acl)
Copied to clipboard
Joshua Ong Jun Leang, Zheng Zhao, Aryo Pradipta Gema, Sohee Yang, Wai-Chung Kwan, Xuanli He, Wenda Li, Pasquale Minervini, Eleonora Giunchiglia, Shay B Cohen
| Challenge: | Recent studies show that large reasoning models (LLMs) achieve strong performance on complex reasoning tasks. |
| Approach: | They propose a method that scores each candidate generation using the joint log-likelihood of the reasoning and final answer. |
| Outcome: | The proposed method outperforms baselines with 2x fewer samples in 20 out of 25 comparisons. |